Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational

As a result, it tractably draws samples even when the matrices forming the Khatri-Rao product have tens of millions of rows each.

Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer.

One such filter, the partitioned learned Bloom filter (PLBF), achieves excellent memory efficiency.